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Executive Summary 


The standard setting process for the New York State Regents Examination in 
Geometry (Common Core) consisted of two events: the Performance Level 
Description Development meeting and the standard setting meeting. The primary goal 
for these meetings was to establish cut scores that operationally define the five 
performance levels: Level 1, Level 2, Level 3, Level 4, and Level 5. The performance 
level designations will be used by local, state, and federal accountability programs and 
are central to communicating with parents, teachers, and the public. This document 
provides a detailed description of the activities held at each meeting. 


On February 18, 2015, the Performance Level Description Development meeting was 
conducted in Albany, New York. The focus of this meeting was the development of 
performance level descriptions (PLDs), which describe the specific knowledge and 
skills of students at each level of performance. Each PLD was designed to describe 
the range of students at that performance level and was used in the subsequent 
standard setting meeting. 


On June 16 and 17, 2015, a standard setting meeting was held. The purpose of this 
meeting was to identify four cut scores that distinguish the five levels of performance 
on the New York State Regents Examination in Geometry (Common Core). Using the 
PLDs articulated in February, panelists followed the Bookmark standard setting 
procedure, which resulted in cut score recommendations that were brought to the New 
York State Education Department (NYSED). 


In this technical report, panelists, materials, methodologies, and results for each 
meeting are presented for the New York State Regents Examination in Geometry 
(Common Core). A preliminary summary of standard setting activities was presented 
to the NYSED the day following the standard setting meeting. This report provides 
final results and additional details documenting the standard setting process and the 
results. 


Performance Level Description Development Meeting 


On February 18, 2015, the New York State Education Department (NYSED) 
conducted a Performance Level Description Development meeting in Albany, New 
York. The meeting was convened to articulate the knowledge and skills expected of 
students at each level of performance, consistent with the policy vision set forth by 
the NYSED. 


At this meeting, panelists were asked to review policy guidelines and content 
standards to generate knowledge and skill statements that describe a student ata 
specific level of performance. An initial training session regarding the overall process 
of standard setting and the specific role that PLDs play within standard setting was 
provided by the lead psychometrician. Using NYSED-approved training materials, 
educators were trained to deconstruct the content standards in terms of cognitive 
complexity and then to align these different aspects of the content standards with 
specific levels of performance. Educators were trained to adhere carefully to the 
cognitive alignment (e.g., depth of knowledge, cognitive complexity, and range of 
skills) with the State’s content standards while keeping the policy decisions in mind. 


Panelists 
A total of 15 Geometry subject-matter expert educators attended the meeting. The 
participants were recruited by the NYSED. 


Method and Procedure 

The PLD meeting began with introductions of NYSED staff and the Data Recognition 
Corporation (DRC) facilitators. The lead DRC psychometrician provided an opening 
training session that included an overview of standard setting and the process by 
which cut scores are determined. The policy decisions associated with the standard 
setting, including the number of performance levels (five) and the associated labels 
for these levels, were reviewed. An explanation of how the PLDs document the 
expected knowledge and skills associated with each performance level was then 
provided. The role that the PLDs play in establishing the cut scores between each 
level of performance was described. 


DRC content experts then described the development of the PLDs, which would use 
the four-step process described below. 


Step 1. Review and Internalize Policy PLDs 
Panelists reviewed the statements that describe the policy vision that the NYSED has 
for the Regents Examination performance levels. Panelists were instructed to use 


this vision as the context for preparing the PLDs. Throughout the day, DRC 
facilitators reminded participants to recall the policy expressed in these statements. 


Step 2. Evaluate Content Standards in Terms of Cognitive Complexity 

DRC facilitators walked panelists through several examples of how to deconstruct the 
content standards in terms of the different levels of cognitive complexity until all 
panelists were oriented to this task. The content standards consist of statements that 
describe knowledge, skills, and performance, which range in terms of cognitive 
complexity; these statements are similar to the kinds of statements that are frequently 
included in PLDs. More importantly, the content standards include statements of 
basic skills that students would display, as well as other skills that require more 
advanced cognitive processing by students. The articulation of different levels of 
cognitive complexity reflected in content standards provides the basis for the 
development of the PLDs. 


Step 3. Align Levels of Cognitive Complexity with Performance Levels 

After a thorough evaluation of each content standard, participants identified specific 
statements that described different levels of cognitive complexity for various 
knowledge and skills specified within the content standards. Participants then 
classified each of these statements in terms of the different performance levels. That 
is, each statement that expressed some level of cognitive complexity within a content 
standard was categorized into the different performance levels (i.e., Level 1, Level 2, 
Level 3, Level 4, or Level 5). Some statements were clearly aligned with a given 
performance level. Other aspects of the content standards did not fall cleanly into a 
specific PLD; these reflected a transition point from one performance level to another. 


DRC facilitators showed participants how to use a coding scheme to reflect which 
statements clearly aligned with specific levels of performance and which statements 
reflected transition points. 


Step 4: Prepare Draft PLDs 

The classification of the cognitive complexity of the content standards and associated 
skills in terms of the performance levels provided a straightforward framework that 
enabled participants to create initial drafts of the Range and Threshold PLDs. Skills 
from the content standards that were clearly associated with a specific performance 
level provided insight into what constituted the performance level for all students in 
that level (i.e., Range PLD). Similarly, the skills that spanned adjacent performance 
levels and were difficult to categorize provided insight into what constituted the 
transition between levels of performance (i.e., Threshold PLD, described below). 


Upon completion of the subject-specific training, each group of panelists was divided 
into small teams. Each team was assigned several content standards. Participants 


then deconstructed each content standard and identified the different statements 
about students being made in the standard in terms of cognitive complexity. 
Particular aspects of the content standards that were easily classified into a given 
performance level essentially formed the basis of the PLDs. Working in these teams, 
participants produced drafts of the PLDs. At the conclusion of this activity, the draft 
PLDs were shared across teams for cross-team discussion and revision. When 
teams encountered knowledge and skills that were difficult to classify into a particular 
level of performance, panelists were asked to document such challenges as 
potentially indicative of transitional knowledge and skills that demarcated the 
threshold between performance levels. 


The drafts produced represented the participants’ conceptualization of the range of 
students in each performance level. After the meeting, DRC, working with the 
NYSED, reviewed and revised the PLDs for clarity and consistency. The end result of 
this meeting was a set of PLDs that clearly defined the level of knowledge and skill 
necessary for each performance level. 


Results 

Copies of the final Geometry PLDs developed at this meeting and revised by DRC 
and the NYSED are provided in Appendix A. These PLDs were used in the 
subsequent standard setting meeting. 


Bookmark Standard Setting Meeting 


A committee of New York State educators was convened on June 16-17, 2015, in 
Albany, New York, to recommend performance standards for the New York State 
Regents Examination in Geometry (Common Core). The Geometry committee 
consisted of 28 educators. 


DRC followed a Bookmark procedure similar to the method originally defined by 
Lewis, Mitzel, and Green (1996). The Bookmark procedure is arguably the standard 
setting method that is most philosophically consistent with criterion-referenced, 
standards-based assessments like the Regents Examinations. This method is 
discussed in detail within the Methods section of this document. 


Panelists 

All panelists (committee members) voluntarily provided demographic information. 
Five table leaders for were identified from the pool of panelists by the NYSED and 
DRC. Table 1 provides a summary of gender representation of panelists. Table 2 
presents a summary of ethnic representation of panelists. Table 3 provides a 
geographic summary of panelists. Table 4 provides a summary of the educational 
background of the committee. 


Table 1. Number of Male and Female Panelists in Committees 


17 
11 


Table 2. Ethnic Composition of the Panelists in Committees 


Table 3. Geographic Locations of Panelists for Standard Setting 


*Participants reported multiple assignments 


Method 

The Bookmark procedure was used to determine recommended cut scores for 
distinguishing performance on the Regents Examination in Geometry (Common Core). 
The Bookmark procedure is an item-based mapping method. It requires panelists to 
determine which items can be successfully answered two-thirds of the time by students 
at the boundaries between adjacent performance levels. The scaled difficulty value that 
separates the items that students at the threshold can answer two-thirds of the time 
from those they cannot answer is the cut score used to distinguish student performance 
into performance levels. The procedure typically involves three components: PLDs, 


ordered item booklets (OIBs), and item maps. Each component is briefly described 
below. 


Performance Level Descriptions (PLDs) 

PLDs are the foundation of standard setting activities because they provide the 
explanation of how student performance differs from one performance level to the next 
(Perie, 2008). In fact, PLDs are of such influence that, in a well-run standard setting 
workshop, they determine the rigor of the performance and thus the decisions made 
about placement of the cut score (Perie, Hess, & Gong, 2008). Moreover, PLDs serve 
multiple purposes in terms of communicating policy, facilitating test development, 
guiding standard setting, and providing score interpretation. Three types of PLDs (Egan, 
Schneider & Ferrara, 2012) are used as an organizing framework for developing PLDs 
for the Regents Examinations: 


e Policy PLD Policy Statements—Policy statements are designed to capture the 
vision that an agency has for its performance levels. They specify the number of 
levels and the names for each level and summarize the expectations of student 
performance for a testing program, including any policy decisions being made at 
particular levels. 

e Range PLDs—Range PLDs are designed to describe the full range of 
performance for examinees at a given performance level. In other words, Range 
PLDs describe the aspects of test content or specific items that are indicative of a 
range of students at a specific performance level. Range PLDs can be 
informative in guiding item and test development as a testing program evolves. 
Range PLDs are also critical in that they are used to articulate a key component 
for standard setting, the Threshold PLDs. Note that the PLD meeting held in 
February was designed to produce Range PLDs. 

e Threshold PLDs—Threshold PLDs (also known as Target PLDs) are designed to 
articulate the transition points between the different ranges of performance 
defined by the Range PLDs. Specifically, Threshold PLDs describe the 
knowledge and skills a student at the border between performance levels should 
know and be able to do. Because they articulate the specific performance that 
distinguishes levels of performance, Threshold PLDs are typically used in 
standard setting activities. Range PLDs and Threshold PLDs are clearly 
interdependent, which necessitates that they be developed in conjunction with 
each other. 


Ultimately, PLDs are designed to describe the competencies of each performance level 
in relation to grade-level content standards while concurrently addressing their different 
functions. PLDs play a critical role in the standard setting process. 


Ordered Item Booklet (OIB) 

Within the Bookmark procedure, participants review the OIB, which is a booklet of the 
items from the operational test that have been ordered from easiest to hardest. Multiple- 
choice items appear along with their answer choices in the OIB, with each item printed 
on a single page. Constructed-response items appear along with their scoring rubrics 
multiple times because each item is worth multiple points. Specifically, each non-zero 
score point for a constructed-response item is presented in the OIB. 


To sequence the items from easiest to hardest, a difficulty estimate for each item must 
be determined. Difficulty estimates supporting Bookmark standard setting are typically 
obtained using item response theory models that express item difficulty and student 
achievement on the same reporting scale. The Rasch measurement model (Rasch, 
1960, 1980) was used to estimate item difficulty for selected-response items on the 
Regents Examinations. The Partial-Credit model (Andrich, 1978) was used to estimate 
item difficulty estimates for each score point for constructed- response models. These 
models are described in more detail below. 


Rasch and Partial-Credit Models 
The Rasch model applicable to dichotomously scored items (MC) can be expressed in 
the most familiar form of the model: 

ohn 


1. Pr(correct | £,,,6,) =———. 
iP 1+e%4 


The probability of success for a person with ability 4, on an item with difficulty 6 is 
determined by the difference between the ability of the student and the difficulty of the 
item. 


With the partial-credit model used for open-ended items, mix is the probability that 
person n will score k on item /. Then, the first threshold for item / is a score of 1 rather 
than a 0, which is the conditional probability of a score of 1, given a score of 0 or 1: 


Tinit 7 exp(Z, -6,) 
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where fis the ability of person n and 6) is the difficulty of the first threshold. The 
expression on the right is identical to the Rasch model for a dichotomous item. The only 
differences are that now z,,, + 7,,;, <1, since more than two response categories are 
provided, and 61, while still the difficulty of the first threshold for item /, is not the 
difficulty of the only threshold for the item. 


For example, with a three-point open-ended item, where a person n must achieve one 
of the four possible scores (0, 1, 2, or 3) on item /, 


Haig + By + Epig + Aig =1 


nil 


These relationships can be rearranged to obtain one general expression for the 
probability of person n scoring x on item /: 


exp )'(B, -6)) 


jal x= 1,...,m 
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If the number of thresholds (m)) is one, the summations in expression (4.) drop out and 
it reduces to expression 1. 


Using the operational response data from a representative sample of test takers, item 
difficulty parameter b was calibrated using WINSTEPS. Within the Rasch model, the 
item difficulty estimate produced by WINSTEPS assumes a 0.50 response probability. 
However, in standard setting, item difficulty estimates are typically computed relative to 
a response probability of two-thirds (i.e., 0.67). For dichotomous items, this required 
adding a factor of 0.69315 to the item difficulty parameters obtained from WINSTEPS to 
account for the increased response probability. 


To obtain difficulty values for each score point within a constructed-response item using 
a two-thirds response probability, it was necessary to estimate the ability level 
associated with getting each score point or above. That is, for a four-point item, the 
ability associated with the likelihood of achieving two points or greater two-thirds of the 
time, three points or greater, and four points are estimated. This computation is done 
algorithmically, using a procedure detailed in Cizek and Bunch (2007). 


After all difficulty estimates associated with a two-thirds response probability were 
computed, the OIB was created by ordering items in sequence of the difficulty 
estimates. Table 5 below includes information about the operational test and the OIB. 
Note that each page of the OIB included an annotation with the scaled difficulty 
estimate, key, and content standard. 


Table 5. Composition of Ordered Item Booklet: Geometry 


Number of Score Number of 
Part Point 
Items OIB Pages 
Range 
Part 1 24 0-1 24 
Part 2 7 0-2 14 
Part 3 3 0-4 12 
Part 4 2 0-6 12 
TOTAL 36 - 62 


Item Map 

The item map provides a corresponding document to the OIB. Essentially, the item map 
consists of information extracted from the OIB and presented in tabular form. The item 
map is presented with one row per item/point. The items/points are presented in 
difficulty sequence from easiest to hardest similar to the OIB. Each row includes the 
following information: 


e Page number in OIB 

e Original position on test form 

e Content/standard identification 

e Correct answer for selected-response items 

e Score point and maximum score point for each constructed-response item 
e Space for notes 


Bookmark Judgment Task 

During a standard setting using the Bookmark procedure, panelists review the test items 
ordered by difficulty from easiest to hardest. Item by item, panelists are asked to judge 
the likelihood that a student at the threshold between performance levels (e.g., the 
student who is just barely at Level 4) would answer the question correctly or achieve a 
particular score on a constructed-response item two-thirds of the time. The panelists are 
reminded throughout the process to use the policy guidance and the associated PLDs 
as the frame of reference. Panelists have typically been given an orienting task to 
become very familiar with the policy decisions and range PLDs in order to help 
articulate the knowledge and skills of students at the threshold. Panelists review the OIB 
information and make judgments for one PLD at a time in a specific sequence. 


The specific judgment task with the Bookmark method requires panelists to evaluate 

whether students at the threshold of a PLD (e.g., just barely at level 4) have a chance of 
answering an item correctly or getting a particular score on a constructed-response item 
at a given response probability. The chance of answering (i.e., the response probability) 
that is typically used within Bookmark standard setting is two-thirds. Panelists are asked 
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to look at each item and evaluate whether a student at the threshold has at least a two- 
thirds chance of getting this item correct. For constructed-response items, the judgment 
task is to determine whether the student at the threshold has at least a two-thirds 
chance of achieving a certain number of points or higher on that item. 


Panelists are instructed to move through the OIB, read each page/item in sequence, 
and evaluate the knowledge and skills as described by the PLDs that are required to 
respond to the item correctly (or to get the score point). Panelists are asked to identify 
the location in the ordered item booklet where the likelihood for a student at a given 
threshold to get an item right drops below the response probability of two-thirds. 
Panelists are asked to place a bookmark between the two items, marking the location 
where this transition occurs for this given threshold. Panelists then begin the process 
again for the next threshold until all thresholds have been bookmarked. This process is 
repeated over multiple rounds, with feedback after each round. 


After each round, panelists have bookmarked pages that identify where in the OIB they 
feel each transition from one performance level to another is located. Given that each 
page within the OIB has an associated difficulty estimate expressed on a common 
metric, panelists have identified a cut score that can be used to distinguish student 
performance into two performance levels. Bookmark placements are translated back 
into the scale of measurement used to estimate item difficulties. The median of these 
difficulty estimates provided by the panelists is the recommended cut score for a given 
performance level. 


Data 

Data used to support these meetings were obtained from representative samples of 
students who had been administered the Regents Examinations immediately prior to the 
standard setting meetings. The samples were drawn to be representative of the typical 
population taking these Regents Examinations during a June administration. In order to 
expedite the production of the standard setting materials, a representative sample was 
selected in advance and processed ahead of remaining State materials. Item difficulty 
values, order item sequence, item maps, and impact data used at the standard setting 
meeting were all compiled using the data from this representative sample. 


A preliminary sample was identified, using test enrollment data with a series of 
stratification values that included gender, ethnicity, English language learner (ELL) 
status, student with disabilities (SWD) status, socio-economic status, need/resource 
capacity (NRC) category, and previous performance on the applicable Regents 
Examination. Schools identified as being included in the sample received different 
answer documents for expedited processing by DRC. Some minor adjustments to the 
preliminary sample were made to account for differences between enrollment 
information and actual test administrations. 
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Summary statistics for the sample versus the population of a typical June administration 
(June 2014, in this case) are reported in Table 6. Note that the differences between the 
sample selected and the typical populations taking the Regents Examinations are 
negligible, suggesting that the information presented to standard setting panelists was 
well estimated. 


Table 6. Sample vs. Population Summary, Geometry 
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Procedure 
The standard setting was completed on June 16 and 17, 2015. The agenda for the 
standard setting meeting can be found in Appendix B. 
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Table Leader Training 

Table leaders arrived the evening before the beginning of the standard setting meeting 
for training. Table-leader training consisted of an overview of the meeting agenda and 
the Bookmark procedure. Samples of materials provided for the standard setting were 
presented, and the role of table leaders was reviewed. Table leaders were to facilitate 
discussion and help participants stay focused at specific stages during the standard 
setting meeting. 


Large Group Training 

After the greetings and initial introductions, Senior Deputy Commissioner Wagner 
provided opening remarks and set the context for the meeting. A highlight of his 
presentation was an overview of the policy decisions associated with each performance 
level. These are shown below in Figure 1. 


Figure 1. Policy Statements for Performance Levels 


Level 5: Exceeds Common Core expectations 


Level 4: Meets Common Core expectations 
(First required for Regents Diploma purposes with the Class of 2022) 
Level 3: Partially meets Common Core expectations 
(Required for current Regents Diploma purposes. We expect comparable 


percentages of students to attain Level 3 or above as do students who pass current 
Regents Exams (2005 Standards) with a score of 65 or above) 


Level 2 (Safety Net): Partially meets Common Core expectations 
(Required for Local Diploma purposes. We expect comparable percentages of 
students to attain Level 2 or above as do students who pass current Regents Exams 
(2005 Standards) with a score of 55 or above) 


Level 1: Does not demonstrate Knowledge and Skills for Level 2 


Following the Senior Deputy Commissioner’s remarks, DRC provided an overview of 
the standard setting methodology. The major components of the Bookmark procedure 
were discussed in detail, including the PLDs and the OIB and its associated item map. 
Two procedures to be implemented within the Bookmark context were presented to the 
panelists. 
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1. Given the policy decision to hold the percentage of students at Level 3 and 
above as well as Level 2 and above to similar levels as those obtained in the 
previous Regents Examinations (see Figure 1), a policy validation exercise would 
be conducted. In particular, the bookmark locations that maintain consistency 
with the previous percentages would be pre-identified for panelists. Panelists 
would be asked to choose one of the pre-identified bookmark locations, 
consistent with the policy directive. Feedback on the bookmark placement would 
be gathered. This exercise would be completed as a single activity, and 
recommended cut scores for these two levels would be incorporated into the 
subsequent standard setting activity. 


2. For the Level 4 and Level 5 cut scores, a traditional Bookmark standard setting 
procedure would be implemented. Results from the first activity would be 
incorporated so that panelists would see impact data for all performance levels. 


There were 28 educator panelists for Geometry; each educator was pre-assigned to 
one of five tables. A table leader had also been pre-assigned to each table. 


Following a break, panelists reviewed the test. The goals of the test review were for 
panelists to get a sense of the student experience in taking the Regents Examination 
and for panelists to preview the test items to be used in the standard setting. A subset 
of the items was identified for panelists to answer and score to ensure that the activity 
was not cursory. Panelists were instructed to review the remaining items. 


Following the test review, DRC content facilitators led a discussion of the Level 4 cut 
score. This discussion asked for knowledge and skill statements describing students at 
the thresholds. Each table, working with several assigned domains of content, identified 
knowledge and skill statements that best described students at the thresholds. The 
synthesis of these statements across tables constituted the Threshold PLD and 
provided a frame of reference for the Bookmark task. Panelists, working in groups, 
repeated this process for Level 5. 


Subject-specific training in the Bookmark standard setting method was then provided. 
The critical objective of the training was to ensure that the panelists understood the task 
being presented to them. Components of the training for panelists included a discussion 
of their role in the process, a detailed description of all steps in the Bookmark method, 
and practice exercises that contained publically available New York State assessment 
items. The point of the practice exercises was to provide hands-on experience with the 
steps and allow panelists to address additional questions that they might have once 
they had practiced. A copy of the training slides is provided in Appendix C. Once 
training was completed, a survey was taken to ensure that all panelists were ready to 
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proceed. All panelists indicated that they understood the task and were ready to 
proceed. 


The policy verification task was first implemented for Level 2 and Level 3. For this task, 
the bookmark locations that resulted in equivalent passing rates relative to the previous 
Regent Examination were pre-identified. That is, a set of bookmark locations where the 
resulting percentage of students at Level 2 and above and Level 3 and above were 
identified. A color-coded item map provided the location of bookmark locations that 
would be consistent with the policy directives. Panelists were instructed to review the 
policy directives and the PLDs and identify which of the potential bookmark locations 
they would recommend. Panelists were reminded that the number of bookmark 
locations consistent with the State policy directives was relatively few and appeared 
early in the test book. Panelists completed a rating form to indicate their selected 
bookmark locations consistent with the State policy and completed a survey to 
demonstrate that they understood the policy verification task. 


Once the policy verification was completed, the standard setting process for Level 4 and 
Level 5 was then started within each room. Three rounds were conducted. Each round 
is described below. 


Round 1. Panelists were asked to identify the last item in the OIB that a threshold 
student at a given performance level would have a two-thirds chance of answering 
correctly. The bookmark location that panelists were to mark in the OIB was the last 
item that a student at the threshold could answer correctly two-thirds of the time; the 
student would not be expected to correctly answer the items that appeared later in the 
booklet. Panelists were asked to consider the knowledge and skills required to respond 
correctly to each progressively more difficult item. Panelists were reminded not to focus 
too much on a single item but to focus on the progression of items instead. Panelists 
were reminded that the OIBs were based on analysis of data selected from a 
representative sample from the June 2015 administration. It was emphasized that the 
work for this round was to be done individually. 


Round 2. Table-level results from round 1 were provided to table leaders. Table-level 
results included the bookmark locations (i.e., pages selected by panelists) for each 
panelist and the median bookmark location for each performance level at the table. The 
panelists were asked to think about how similar their ratings were relative to the other 
panelists at their tables. Table leaders facilitated group discussion about 
differences/similarities, using the table-level results. Panelists were reminded that 
consensus was not a requirement and that differences should be discussed in order to 
provide additional insight into why such differences existed. After the group discussion, 
panelists were given the opportunity to revise their bookmark placements in a 
subsequent round. 
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Round 3. Table-level results from round 2 were provided to table leaders. Table-level 
results included the bookmark locations (i.e., pages selected by panelists) for each 
panelist and the median bookmark location for each performance level at the table. The 
panelists were asked to think about how similar their ratings were relative to the other 
panelists at their tables. Table leaders facilitated group discussion about 
differences/similarities, using the table-level results. Panelists were reminded that 
consensus was not a requirement and that differences should be discussed in order to 
provide additional insight into why such differences existed. 


After table discussions were complete, the DRC facilitator presented table-level results 
as well as the room-level results to the full group. In particular, the median bookmark 
locations for all tables, as well as the room-level median of table-level medians, were 
presented. Panelists were then invited to discuss the table-level and room-level results, 
comparing and contrasting differences between tables and providing their initial 
feedback regarding the room-level results. 


Once discussion of the table-level and room-level bookmark locations was complete, 
impact data based on the representative sample were provided to panelists. 
Specifically, the percentages of students at the different levels of performance were 
provided to panelists. After this discussion was complete, panelists were given another 
opportunity to revise their bookmarks. 


After round 3 rating and analysis were completed. The final recommendations for 
bookmark locations, as well as the associated impact data, were presented. Panelists 
were also invited to provide any additional feedback about the PLD documents. 


Results 

Tables 7 through 9 provide summary information for all performance levels for 
Geometry across all three rounds of standard setting. The median bookmarked page for 
each table and the associated median difficulty estimate are provided. The difficulty 
estimate is based on a two-thirds response probability and is expressed on the logit 
scale used within the Rasch model. The room-level summary, computed as the median 
of table-level medians, is also presented. Because only one round was held for the 
policy verification of Levels 2 and 3, that information is repeated throughout the tables 
so that the results across all four levels can be compared. 
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Table 7. Median Bookmarked Pages, Geometry, Round 1 


Level 1/2 Level 2/3 Level 3/4 Level 4/5 


Median 
OIB 


49 —s:1.4346 


Level 1/2 Level 2/3 Level 3/4 Level 4/5 


7 


 -0.7737 0.1917 34 =0.9409 49 1.4346 
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Comparisons between rounds also indicate that the cut score recommendation did not 
fluctuate much. Impact data were presented at the beginning of round 3. The additional 
information had a negligible effect on the subsequent recommendations that the group 
made in round 3. 


Figures 2 and 3 represent the percentage of students in each performance level, using 
the cut score recommendation after rounds 2 and 3 for Geometry. The impact data were 
based on a representative sample of students who were administered the 2015 Regents 
Examination. Note that these were the figures that were used to present impact data to 
panelists. 
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Figure 2. Percentage of Students in Performance Levels, Geometry, Round 2. 


Round 2 Impact, Geometry 


Figure 3. Percentage of Students in Performance Levels, Geometry, Round 3 


Round 3 Impact, Geometry 


@ Level 1 
B Level 2 
m Level 3 
B Level 4 


@Level5 


Level 1 


Level 2 


@ Level 3 


mw Level 4 


@Level5 
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Evaluations 

An exit survey was completed by each panelist after the policy verification of the Level 2 
and Level 3 cut scores. Panelists answered the survey questions using a scale of 1-4, 
with 1 being “strongly disagree” and 4 being “strongly agree.” The survey questions and 
the results for each question are provided in Appendix D. 


The intentions of this exit Survey were to ensure that panelists understood the policy 
directives to place constraints on the overall standard setting process and to get their 
feedback about the recommended cut scores, given the policy directives. Over 95% of 
the panelists moderately or strongly agreed that they understood the policy directives 
and that the projected bookmarks fairly represented the minimal level of achievement 
for students at Level 2 and Level 3. 


An additional exit survey was completed by each panelist after all standard setting 
activities were completed. Panelists answered the survey questions using a scale of 
1-4, with 1 being “strongly disagree” and 4 being “strongly agree.” The survey questions 
and the results for each question are provided in Appendix E. 


The intentions of this exit survey were to gather feedback on different aspects of the 
standard setting procedure and to get panelists’ feedback on the recommended cut 
scores and associated results. All of the panelists moderately or strongly agreed that 
the cut scores accurately represented the PLDs. All of the panelists felt that the 
Bookmark standard setting method and associated activities would produce appropriate 
results for New York State students. 


Final Recommendations 

As described in the previous sections, the NYSED, with facilitation by DRC, conducted 
a formal standard setting that consisted of two meetings. The first meeting was devoted 
to the development of PLDs that articulate the range of knowledge, skills, and 
proficiencies of students at the five levels of performance specified by State policy. The 
second meeting was dedicated to the identification of cut scores consistent with the 
PLDs and State policy directives, using a standardized, scientific procedure called the 
Bookmark method. 


Both meetings reflected best psychometric practices as articulated in the Standards for 
Educational and Psychological Measurement, and proceeded according to the plans 
reviewed by the New York State Technical Advisory Committee. The participants in both 
meetings were diverse and representative of the State. All groups followed, without 
incident, instructions delivered by standard setting staff. All activities were formally 
overseen by the Office of State Assessment senior management and psychometric 
Staff. 
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After careful consideration of the nature of the new examinations, the rigor of the new 
curricula, the transitional and aspirational aspects of the State policy directives, and the 
role of the assessment in student learning throughout high school and beyond, the 
standard setting committees made recommendations on the cut scores to the 
Commissioner of Education. The Commissioner accepted the recommendations of the 
standard setting panelists. The approved cut scores were provided to the NYSED’s 
scaling and equating contractor for implementation within the scale of measurement 
used to report student performance on the New York State Regents Examinations. 


The standard setting process was developed and implemented with great care, and 
best practices in assessment and psychometrics were followed. The policy decisions 
implemented were consistent with sound psychometric research to guarantee an 
effective and efficient standard setting. 
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Appendix A: Range Performance Level Descriptions, Geometry 


Domain NYS Level 5 NYS Level 4 NYS Level 3 NYS Level 2 NYS Level 1 
Congruence Use precise language | Describe a sequence Identify and draw a Identify and draw a Sketch triangles 
(G-CO) to describe a sequence | of rigid motions to sequence of rigid rigid motion in the and rectangles. 


of rigid motions to 
determine the 
congruency of figures. 


determine the 
congruency of 
figures. 


motions in the plane 
to verify the 
congruency of 
figures. 


plane. 


Use precise language 
to predict the effect of 
a given rigid motion 
on a given figure. 


Predict the effect of a 
given rigid motion on 
a given figure. 


Identify the image and 
describe the effect of 
a given rigid motion. 


Identify the image of 
a given rigid motion. 


Formulate a complete 
line of geometric 
reasoning to prove a 
geometric theorem. 


Formulate a 
complete line of 
geometric reasoning to 
prove a specific 
geometric statement. 


Formulate a partial 
line of geometric 
reasoning in an effort 
to prove a specific 
geometric statement. 


Provide a correct 
geometric statement 
pertaining to the 
given geometric 
information. 


Restate given 
information in the 
context of a 
proof. 


Use the rotations and 
reflections that carry a 
figure onto itself to 
prove or explain if the 
figure is or is not 
regular. 


Describe the rotations 
and reflections that 
carry a figure onto 
itself. 


Identify the rotations 
and reflections that 
carry a figure onto 
itself. 


Determine the validity 
of geometric 
arguments and revise 
invalid geometric 
arguments. 


Determine the validity 
of geometric 
arguments with 
justification. 


Determine the validity 
of geometric 
arguments. 
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Domain NYS Level 5 NYS Level 4 NYS Level 3 NYS Level 2 NYS Level 1 
(G-CO Make advanced Construct the Make basic formal Construct rays, Construct lines 
continued) formal geometric application of the geometric triangles, and angles. | and line 

constructions using listed constructions, | constructions using segments. 
appropriate tools. for example, using the | appropriate tools. 

construction of a Examples of basic 

midpoint to construct | constructions include 

the median of a but are not limited to: 

triangle or construct copy a segment, 

the dilation of a figure | bisecting a segment, 

not on the coordinate _| bisecting an angle. 

plane. 

Domain NYS Level 5 NYS Level 4 NYS Level 3 NYS Level 2 NYS Level 1 
Similarity, Use precise language | Describe a sequence | Identify a sequence Perform a dilation in 
Right to describe a sequence | of similarity of similarity the coordinate plane 
Triangles, of similarity transformations to transformations in centered at the origin. 
and transformations to determine the the plane to verify the | Distinguish between 
Trigonometry | determine the similarity of figures. | similarity of figures. | a dilation anda 
(G-SRT) similarity of figures. translation, reflection, 


or rotation. 


Formulate a complete 
line of geometric 
reasoning to prove a 
geometric theorem. 


Formulate a 
complete line of 
geometric reasoning 
to prove a specific 
geometric statement. 


Formulate a partial 
line of geometric 
reasoning in an effort 
to prove a specific 
geometric statement. 


Provide correct 
geometric statements 
pertaining to the 
given geometric 
information. 


Restate given 
information in 
the context of a 
proof. 
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Domain NYS Level 5 NYS Level 4 NYS Level 3 NYS Level 2 NYS Level 1 
Apply congruence or | Apply congruence or | Apply congruence or | Apply congruence or 
(G-SRT similarity criteria to similarity criteria to similarity criteria to similarity criteria to 
continued) solve complex solve problems, and solve problems. solve simple 
problems involving explain the problems. 
multiple concepts, geometric reasoning 
and explain the involved. 
geometric reasoning 
involved. 
Use the Pythagorean | Use the Pythagorean | Identify the Sketch and label 
Theorem, Theorem, trigonometric ratios | the sides of right 
trigonometric ratios, trigonometric ratios, | of a right triangle. triangles. 


and the relationship 
between sine and 
cosine of 
complementary angles 
to solve complex 
problems. 


and the relationship 
between sine and 
cosine of 
complementary 
angles to solve 
problems. 


Determine the validity 
of geometric 
arguments and revise 
invalid geometric 
arguments. 


Determine the validity 
of geometric 
arguments with 
justification. 


Determine the validity 
of geometric 
arguments. 
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Domain NYS Level 5 NYS Level 4 NYS Level 3 NYS Level 2 NYS Level 1 
Circles Use appropriate tools | Use appropriate tools | Use appropriate tools 
(G-C) to construct the to construct the to construct the 
inscribed and inscribed and circumscribed circle 
circumscribed circle circumscribed circle for a given triangle. 
for a given triangle for a given triangle. 
and justify the 
construction. 
Derive the formula Apply formulas for Determine the arc Determine the area of | Write an 


for the arc length and 
area of a sector. 


arc length and area of 
a sector to solve 
complex problems. 


length and area of a 
sector given any 
central angle in 
degrees or radians. 


a quarter, half, or 
three-quarter circle, 
given the area of the 
entire circle. 


expression for the 
area of a circle 
given the radius. 


Apply theorems about 
arcs, angles, and 
segments related to 
circles. 


Apply theorems 
about arcs and angles 
related to circles. 


Identify arcs, angles, 
and segments related 
to circles. 


Visually compare 
central angle 
measures. 


Explain the radian 
measure of a central 
angle as the constant 
of proportionality 
between the arc length 
and the radius of a 
circle. 


Identify central angles 
in different circles that 
have the same radian 
measure. 
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Domain NYS Level 5 NYS Level 4 NYS Level 3 NYS Level 2 NYS Level 1 
(G-C Formulate a Formulate a partial Identify a missing 
continued) complete line of line of geometric angle in a diagram 


geometric reasoning to 
prove properties of 
angles fora 
quadrilateral inscribed 
in a circle. 


reasoning in an effort 
to prove properties of 
angles fora 
quadrilateral inscribed 
in a circle. 


involving a 
quadrilateral 
inscribed in a circle. 


Formulate a 
complete line of 
geometric reasoning to 
prove that circles are 
similar. 


Formulate a partial 
line of geometric 
reasoning in an effort 
to prove that circles 
are similar. 


Find missing radius 
and circumference 
measurements using 
circle similarity. 
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Domain NYS Level 5 NYS Level 4 NYS Level 3 NYS Level 2 NYS Level 1 
Expressing Use the Pythagorean | Given the equation of | Identify the center Complete the square 
Geometric theorem to derive the | a circle in standard and radius of a circle | witha single 
Properties equation of acircle. | form, complete the when given the variable. 
with square to obtain the | equation in center- 
Equations center and radius. radius form. 
(G-GPE) Use coordinates to Use coordinates to Use numerical Given three 
formulate acomplete | formulate a coordinates to coordinates of a 
line of geometric complete line of formulate a partial line | special quadrilateral, 
reasoning to prove or | geometric reasoning | of geometric reasoning | determine the fourth 
disprove a geometric | to prove a specific in an effort to prove a_| coordinate. 
theorem. geometric statement. | specific geometric 
statement. 
Explain why parallel | Use the slope criteria | Identify the equations | Identify the slope of a | Distinguish 
lines have the same for parallel and of lines as parallel, line given its between lines in a 
slopes and perpendicular lines to | perpendicular, or equation. coordinate plane 
perpendicular lines solve geometric neither. with positive and 
have negative problems. negative slopes. 
reciprocal slopes. 
Identify the rational | Identify the whole Identify the Locate the 


coordinates of a point 
that divides a segment 
into a given ratio. 


number coordinates 
of a point that divides 
a segment into a 
given ratio. 


coordinates of the 
midpoint of a line 
segment. 


midpoint of a 
horizontal or 
vertical line in a 
coordinate plane. 
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Domain NYS Level 5 NYS Level 4 NYS Level 3 NYS Level 2 NYS Level 1 
(G-GPE Use coordinates to Use coordinates to Use coordinates to Compute the length | Identify the 
continued) compute perimeters compute perimeters of | compute perimeters | of vertical, whole number 


and areas of 
compound figures. 


polygons and areas of 
triangles and 
rectangles with 
rational or irrational 
bases and heights. 


of polygons with 


rational side lengths. 


Use coordinates to 
compute areas of 
triangles and 
rectangles with 
rational bases and 
heights. 


horizontal, and 
diagonal segments on 
the coordinate plane 
with integer 
coordinates. 

Compute the 
perimeter of 
polygons with integer 
side lengths in the 
coordinate plane. 


Compute the area of 
triangles and 
rectangles with 
integer bases and 
heights in the 
coordinate plane. 


coordinates of 
triangles and 
rectangles. 
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Domain NYS Level 5 NYS Level 4 NYS Level 3 NYS Level 2 NYS Level 1 
Geometric Write a formal Write an informal 
Measurement | argument for the argument for the 
& Dimensions | formulas for the formulas for the 
(G-GMD) circumference of a circumference of a 
circle, area of a circle, | circle, area of a circle, 
and volumes of a and volumes of a 
cylinder, pyramid, and | cylinder, pyramid, and 
cone. cone. 
Use the volume Use the volume Use the volume Compute the Compute the 
formulas for cylinders, | formulas for cylinders, | formulas for volumes for volume of a 


pyramids, cones, and 
spheres to solve 
modeling problems 
involving compound 
figures. 


pyramids, cones, and 
spheres to solve 
modeling problems. 


cylinders, pyramids, 
cones, and spheres to 
find various 
dimensions of the 
solid, such as finding 
the radius of a sphere 
given the volume. 


cylinders, cones, and 
spheres. 


rectangular prism 
with integer 
dimensions. 
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Domain NYS Level 5 NYS Level 4 NYS Level 3 NYS Level 2 NYS Level 1 
(G-GMD Describe the Describe the two- Identify the two- Identify a two- Identify the shape 
continued) similarities and dimensional cross- dimensional cross- dimensional cross- of the base of a 


differences between 
various cross- 
sections of three- 
dimensional objects, 
such as explaining the 
difference between the 
areas of different 
cross-sections of the 
same figure. 


sections of three- 
dimensional objects. 


sections using a 
diagram of a three- 
dimensional object. 


section that results 
from slicing a right 
rectangular prism or 
a right rectangular 
pyramid. 


rectangular prism, 
triangular prism, 
or cylinder. 


Describe the 
similarities and 
differences between 
various rotations of 
two-dimensional 
objects, such as a half 
rotation or rotating 
about different axes. 


Describe three- 
dimensional objects 
generated by rotations 
of two-dimensional 
objects. 


Identify three- 
dimensional objects 
generated by rotations 
of two-dimensional 
objects. 
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Domain NYS Level 5 NYS Level 4 NYS Level 3 NYS Level 2 NYS Level 1 
Modeling Create a model to Apply geometric Apply concepts of Given two of the Compute the area 
with solve real-world concepts in modeling | density to solve a three values inthe | of a rectangular 
Geometry problems, which may | situations to solve problem that may density formula, region, given 
(G-MG) include applying complex real-world | include converting find the third value. | whole number 


density to real-world 
situations or solving 
design problems. 


problems, which may 
include applying 
density to real-world 
situations or solving 
design problems. 


between two- and 
three-dimensional 
units. 


dimensions. 
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Appendix B: Agenda for Standard Setting 


NEW YORK STATE REGENTS EXAMINATIONS IN GEOMETRY (COMMON CORE) 
STANDARD SETTING 
JUNE 16-17, 2015 


AGENDA 


Tuesday, June 16, 2015 


8:00 am — 8:30 am Registration and Breakfast 

8:30 am —10315 am Welcome and Training 
The purpose of the day’s first session is to provide background 
information on this standard setting meeting and articulate your roles 
and responsibilities in the standard setting process. A detailed 
overview of the process being used will be given. 

10:15 am — 10:30 am Break 

10:30 am — 11:45 pm Introductions and Test Review 
The goal of the test review is to review the operational test 
individually to get a sense of the student experience and to preview 
the test items that will be used in the bookmark process. 

11:45 pm —12:45 pm Lunch 

12:45 pm — 2:45 pm Review PLDs and Discuss Threshold Students 
The goal of this discussion is to develop a common understanding of 
the students at each threshold and to articulate a description of 
students at the thresholds. 

2:45 pm — 3:00 pm Break 

3:00 pm — 3:30 pm Refresher Training 


The purpose of this training session is to re-orient you to the 
bookmark method and go through a practice activity. 


3:30 pm — 4:00 pm Level 3 and Level 2 Bookmark Evaluation 


The purpose of this session is to evaluate the policy directive in 
relation to the Level 3/Level 2 and Level 2/Level | cut scores. 


4:00 pm— 5:00 pm Level 4 and Level 5 Bookmarking - Round 1 
During round 1, you will individually determine the bookmark 
placement for the thresholds based on the threshold PLDs and your 


professional expertise. These bookmark placements will be translated 
to cut scores for the exam. 


(over) 
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Wednesday, June 17, 2015 


8:00 am — 8:30 am 


8:30 am — 10:00am 


10:00 am — 10:15 am 


10:15 am —11:30 am 


11:30 am — 12:30 pm 


12:30 pm — 2:00 pm 


2:00 pm — 3:30 pm 


3:30 pm — 4:00 pm 


4:00 pm — 4:30 pm 


Breakfast 
Discussion of Round 1 Results 
The goal of this session is to discuss and gain perspective of table 


peers regarding round | bookmark placements. A consensus does not 
need to be reached. 


Break 
Level 4 and Level 5 Bookmarking - Round 2 


During round 2, you will individually determine the bookmark 
placement for the thresholds based on the threshold PLDs and your 
professional expertise. These bookmark placements will be translated 
to cut scores for the exam. 


Lunch 

Discussion of Round 2 Results 
The goal of this session is to discuss and gain perspective of all 
subject peers regarding round 2 bookmark placements. The room 


facilitator will share overall recommended bookmark cut scores as 
well as impact data based on the cut scores. 


Level 4 and Level 5 Bookmarking - Round 3 
During round 3, you will individually determine the bookmark 
placement for the thresholds based on the threshold PLDs and your 
professional expertise. These bookmark placements will be translated 
to cut scores for the exam. 

Break 

Discussion of Round 3 Results 
Final impact results based on the recommended cut scores will be 


shared and reactions to the bookmark process and impact results will 
be discussed. 


34 


Appendix C: Training Slides 


engage” 


Students Their Moment. 


New York State Regents Examination in 
Geometry (Common Core) Standard Setting 


Albany, New York 
June 16-17, 2015 


Opening Session Agenda 


+ Welcome 

+ NYSED Remarks 

+ Purpose of Meeting 

* Standard Setting Overview 
* Methodology 

+ Administrative Issues 


What is Standard Setting? 


+ A process that lets experts make judgments 
about the content that students at each level 
of performance should know 


Welcome and Introductions 


+ Data Recognition Corporation 
o Marc Julian, Trainer and Lead Facilitator 
o Pete Tressel, Project Manager 
o Ben Sorenson, Data Analyst 
o John Selisky, Geometry Content Specialist 


Welcome and Introductions 


New York State Department of Education 
o Ken Wagner, Senior Deputy Commissioner 
o Candy Shyer, Assistant Commissioner 
o Zach Warner, Project Coordinator 
o Paul Anderson, Geometry Lead 


Performance Levels 


Level 5: Exceeds Common Core expectations 


Level 4: Meets Common Core expectations 


(First required for Regents Diploma purposes with the Class of 2022) 


Level 3: Partially meets Common Core expectations 
(Required for current Regents Diploma purposes. We expect comparable 
percentages of students to attain Level 3 or above as do students who pass current 
Regents Exams (2005 Standards) with a score of 65 or above) 


Level 2 (Safety Net): Partially meets Common Core expectations 
(Required for Local Diploma purposes. We expect comparable percentages of 


students to attain Level 2 or above as do students who pass current Regents Exams 
(2005 Standards) with a score of 55 or above) 


Level 1: Does not demonstrate Knowledge and Skills for Level 2 
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engage” 


Our Students. The Moment. 


New York State Regents Examination in 
Geometry (Common Core) Standard Setting 


Albany, New York 
June 16-17, 2015 


How are Standards Set? 


sabia tidal Abadi: 


' 
Levelt {  Level2 
H 
i 


' 
Level 3 H Level 4 Level 5 
H 
H 


Level 2 
Cut 
Score 


Policy Statements 


setitlaateabibtth had 


' 
\ i 

Levellf Level 2 } level 3 | Level 4 }  Level’S 
i 


* Broad statements that define level of rigor and/or 
policy implication for each performance level 


Level 4: Meets Common Core expectations 
(First required for Regents Diploma purposes with the Class of 2022) 
Level 3: Partially meets Common Core expectations 
(Required for current Regents Diploma purposes. We expect comparable 
percentages of students to attain Level 3 or above as do students who pass current 
Regents Exams (2005 Standards) with a score of 65 or above) 


Why do Standard Setting? 


+ We need to decide what score on the Regents 
exams tells us whether a student’s 
performance was enough to be considered 
“Level 4” 


Phases of Standard Setting 


Performance Levels 


Level 5: Exceeds Common Core expectations 


Level 4: Meets Common Core expectations 
(First required for Regents Diploma purposes with the Class of 2022) 


Level 3: Partially meets Common Core expectations 
(Required for current Regents Diploma purposes. We expect comparable 
percentages of students to attain Level 3 or above as do students who pass current 
Regents Exams (2005 Standards) with a score of 65 or above) 


Level 2 (Safety Net): Partially meets Common Core expectations 
(Required for Local Diploma purposes. We expect comparable percentages of 
‘students to attain Level 2 or above as do students who pass current Regents Exams 
(2005 Standards) with a score of 55 or above) 


Level 1: Does not demonstrate Knowledge and Skills for Level 2 
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How are Standards Set? 


iui hadiiti dita: 


' 
Level 1 Level 2 H Level 3 Level 4 Level 5 
i 


Phases of Standard Setting 


' 
Level 2 Level 3 
Cut Cut 
Score Score 


Reflect Current 
Cut Scores 


Performance Level Descriptions Conceptualize the Threshold Student 


HR A HT ARAM 


Level 1 Level 2 Level 3 Level 4 Level 1 Level 2 Level 3 Level 5 
Lower Higher 
* Performance Level Descriptions: 
Describe the knowledge and skills expected of students who are: 
students at each of the five performance levels 3 
P o Just barely entering the next higher level 
Describe the range of performance of the level, not What k led dakills shouldtacatidanth 
the boundary between levels ° at knowledge and skills should a student have 


t the thresholds? 
Developed in previous meetings by NYS educators ee eee 


« Based on the PLDs, visualize New York State 


Phases of Standard Setting Standard Setting Method 


* Bookmark Procedure 
o Focuses on items 


o Focuses on students at the threshold between 
performance levels 

o Focuses on what a student should know and be 
able to do, as defined by the performance level 
descriptions (PLDs) 


Overview of Method Take the test 
Take the test 
Review performance level descriptions * Objectives 
Develop and review threshold students o Take a sample of test items 


descriptions Provide you with insight into the student testing 


. experience 
Multiple rounds of bookmark placements Thinkeebout hawetudents would have 


o Work individually experienced the test 
o Group discussion during 24 and 34 What knowledge and skills does each item 
o Feedback after each round require? 


Ordered Item Booklet Ordered Item Booklet 


+ Items are ordered by difficulty 


o Easy items in front; hard items in 
back 


+ One MCI/CR point per page 
o CRs appear multiple times 
o Scoring rubric for each 
CR score point 
+ Difficulty estimates based ona * OIB for 6 item test 
sample of NYS students that is o 5MC items, 1 CR item worth 3 points 
representative of a typical o 8 Pages inthe OIB 
Regents Exam administration o Answer key, scaled difficulty value, and CR rubric 


Ordered Item Booklet Ordered Item Booklet 


Easier 
+ Key Task + Key Point 
o Place a bookmark between items that mark the o Do not focus ona single item 


transition between one level to another o Identify groups of items where the transition 
o Do that for each threshold occurs 


How are Standards Set? 


Outcomes 


} fdiaditidhtath * This committee is making a set of 
} recommendations 


Level 1 Level 2 Level 3 Level 4 


Current Diploma Reflect New 
Cut Scores Expectations 


Administrative Issues 


* Roles and Responsibilities 
o NYSED Representatives 
Project Managers 
Room Facilitators 
Content Specialists 
Table Leaders 
Panelists 


Administrative Issues 


+ Other meeting guidelines 
No cell phones, tablets, or laptops at tables 
Sit quietly if others are still working 
No materials leave the room 
Start on time 
Participate! 


+ Cut scores for these performance levels will be 
recommended to the Acting Commissioner of 
Education, subject to the approval of the 
Board of Regents 


Level 5 


Administrative Issues 


* Security 
o Non-disclosure agreement 
o All materials stay in room 
o Notify the room facilitator for unscheduled breaks 


o Discussion outside of room 
+ Do NOT discuss: 
— Results, PLDs, Items 
+ DO discuss process 


Forms and Documents 


Non-Disclosure Agreement 

Travel Expense Reimbursement Form 
Substitute Teacher Reimbursement Form 
Panelist Survey 
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Next Steps 


+ Complete forms and give to DRC staff: 
o Non-disclosure agreement 
o Panelist survey 

* Take a 15 minute break and return here 


Questions? 
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engage Today’s Agenda 
Introductions 

New York State Regents Examination in Geometry Take the Test 


(Common Core) Standard Setting fitch 


Review PLDs and Create Threshold 
Descriptions 

Training 

Level 3 and Level 2 Review 

Level 4 and Level 5 Bookmarking - Round 1 


Albany, New York 
June 16-17, 2015 


Introductions Take the Test 


+ What is your name? Please take the following items on the test: 

+ Where are you from? o MC —4, 7, 8, 11, 14, 15, 17, 20, 22, 24 

+ What is your current role in education? o CR-27, 28, 29, 30, 34, 35, 36 
Please review remaining items if you have time 
When done, please sign your test book in 
Report back to this room after lunch 


LUNCH Threshold Student Descriptions 


* Served in Great American Grill For each threshold: 

+ Meet back here at 12:45pm Review performance level descriptions 
Generate knowledge and skill statements 
Develop summary of knowledge and skill 
statements for threshold students 


In the following sequence 
+ Level 3/4 threshold 
+ Level 4/5 threshold 


Review the PLDs 


Wa, AARC RR RAM 


Level 1 Level 2 Level 3 Level 4 Level 5 


* Performance Level Descriptions: 
o Describe the knowledge and skills expected of 
students at each of the five performance levels 
o Describe the range of performance of the level, not 
the boundary between levels 


Conceptualize the Threshold Student 


Level 4: Meets Common Core expectations 
(First required for Regents Diploma purposes with the Class of 2022) 


Divide the Geometry domains among the tables 
At each table, develop knowledge and skill statements for each 
threshold 


>  Operationalize each statement for the threshold students — i.e. what 
would be “just enough” or “just barely” sufficient for each PLD 
‘statement 


Consider behaviors and classroom experiences directly linked 
to the PLDs 


Focus on knowledge and skills 

» Avoid other students attributes (e.g., low SES) 
Form a group definition (concept) of a threshold student 
discussing across tables 


Break 


+ Return to this room at 3pm 


Conceptualize the Threshold Student 


Level 1 Level 2 Level 3 Level 4 Level 5 
Lower Higher 


+ Based on the PLDs, visualize New York State 
students who are: 
Just barely entering the next higher level 
What knowledge and skills should a student have 
at the thresholds? 
We are focusing on the thresholds between 
performance levels that reflect new expectations 


Conceptualize the Threshold Student 


+ Divide the Geometry domains among the tables 


+ Ateach table, develop knowledge and skill statements for each 
threshold 


»  Operationalize each statement for the threshold students — i.e, what 
would be “just enough” or “just barely” sufficient for each PLD 
statement 


Consider behaviors and classroom experiences directly linked 
to the PLDs 
Focus on knowledge and skills 

» Avoid other students attributes (e.g., low SES) 


Form a group definition (concept) of a threshold student 
discussing across tables 


Bookmark 
Training 
and Practice 
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Relationship of PLDs, Performance 
Levels and Cut Scores 


tes =a 

<a> eee ==, 
<> (eee SS I 
ey a ee a | 
ap a 


Ordered Item Booklet 


* Items are ordered by difficulty 
o Easy items in front; hard items i 
back 
+ One MC/CR point per page 
o CRs appear multiple times 
o Scoring rubric for each 
CR score point 
+ Difficulty estimates based ona 
sample of NYS students that is 
representative of a typical 
Regents Exam administration 


OIB — Constructed-Response Item 


Item Image 


Item Information 


OIB Page Number —> 


a 


Materials 


PLDs 


+ Description of Threshold Students 
- OIB 
* Item Map 


Item Separation Chart 
Bookmark Form 


OIB - Multiple-Choice Item 


i OIB Page Number 


| <€— Item information 


Item Map 
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Item Separation Chart 


New York Stale Regente Exarnination in Georrebyy (Common Cored Training 
tam Separation Chact 


) a 
ss 


Placing a Bookmark 


+ Go through OIB page by page 


+ Judge whether threshold 


students should be able to get it 
correct or score that CR point 


or higher 
o Use 2/3rds as criterion 


Placing a Bookmark 


+ Place Bookmark: 

o Your bookmark will be 
between 2 pages 
Place a post-it on the 
last item where you 
think the threshold 
student should get 
the item correct 
2/3rds of the time 
Record this page 
number on your 
bookmark form 


“ill 


Bookmark Form 


OIB Page Number 


‘Your “bookmark” will be between 2 pages — write down the first page. (This 
will be the last item the threstiold student should be able to answer correctly 
U3 of the tin 


Placing a Bookmark 


Key Task: 

o Continue until you reach a point 
where the threshold student 
would consistently not answer 
the item correctly at least 
2/3rds of the time 

Key Points 

o Do not focus ona single 
item 
Identify groups of items 
where the transition 
occurs 


Important Points 


Regents Exams are constructed using the statistical 
model called item response theory (IRT). All state 
testing programs and most certification tests use IRT 
IRT allows different test forms with different items to 
have scale scores with the same meaning within the 
same subject area (e.g., June and August Geometry 
Regents Exams) 

With IRT, scaled scores are designed to award points 
for not just how many items a student gets correct, but 
the challenge-level of the item 

Our raw-to-scale conversion charts reflect this and the 
bookmark standard setting method is designed with IRT 
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Important Points 


The ordered item booklets are based on IRT 
Each item is a representation of the knowledge 
and skills required to achieve a particular scaled 
score 

The page number is not the same as the raw 
score (number correct/points achieved) 
Thinking about bookmark pages as if they were 
raw scores is inconsistent with the bookmark 
methodology 


Remember 


+ Do not consider the page number as a proxy for 
number of points achieved on the test! 


Practice Exercise 


+ For each item, indicate on the item map or the 
OIB if you expect the threshold student to 
answer the item correctly at least 2/3rds of the 
time (Y) or less than 2/3rds (N) 

Place a post-it note on the last item you judge 


that your threshold student would get correct at 


least 2/3rds of the time 

Indicate on the training bookmark placement 
form the last item you judge your threshold 
student would get correct at least 2/3rds of the 
time 


Example 


gota 


Practice Exercise 


+ Review the sample Geometry items 
+ Using the threshold descriptions that we 
created, visualize a student just barely out of 
the Level 3 and just barely into Level 4 
Go through the training OIB page by page 
o assess whether a just barely Level 4 student has 
a sufficient probability of answering each item 
correctly 
* Sufficient is defined as 2/3rds of the time 


Practice 
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Practice Exercise: Sample Results 


* After round 1; 
> Individual cut scores (i.e., 
OIB page) given to each 
table leader 


> Table cut score (median) 

provided to each table 
+ After round 2: 

> Individual and table cut 
scores provided to each 
table 
Table cut scores and overall 
cut score recommendation 
shown to entire room 


Practice Exercise: Sample Results 


Round 3 impacts 


os ee ee ee ee _ oe oe 


After rounds 2 and 3, data will be provided 
showing the percent of students that would be 
in each performance level based on the 
recommended cut scores 


Reflects Current 
Diploma Cut Scores 


Practice Exercise: Sample Results 


* After rounds 2 and 3, impact data across 
rounds will be presented 


Questions? 


f int bata ditt Ai ety 


Levelt | Level2 $ Level3 | Level4 | Level 


Level 2 Level 3 
Cut Cut 
“e Fa 
Must be consistent with 
current passing rates 


* We provide the location of the bookmarks in 
the OIB that are consistent with this policy 

+ We provide a narrow range of items where the 
bookmarks can be moved that maintain the 
policy 
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past ibd aa ada 


Levelt | Level2 {| Levels | — Level4 | Level 


Must be consistent with 
current passing rates 


+ Your task 


Review narrow range of items around the bookmark 
Decide whether to move the bookmark within the 
range based on your expert judgment and PLD 
discussions 


Does not require multiple rounds 


ngageNY.org 


k in ibaa Andy 


Level 1 ' Level2} Level 3 Level4 | Level 5 


+ Your task for these thresholds is to 


o Review each item in sequence and ask yourself 
whether a student at a given threshold would get an 
item right most of the time (2/3rds) 


o Identify where in the OIB the answer to that question 
transitions from Yes to No 


o Multiple rounds with feedback after each round 


Policy Review: 
Level 3 and Level 2 


pagt bitin adahdat th dh dey 


Levelt | Level2 | Level3 | Level4 | Level 


* Policy Decisions for upper 2 cut scores 


o Level 4 and Level 5 cut scores reflect new 
expectations for students 


o By 2022, the Level 4 cut score will be required for 
a Regents diploma 


Questions? 


+ Complete Readiness Form 


Policy Review: Level 3 and Level 2 


Policy directive requires that the percent of 
students at or above these two levels should 
be consistent with current levels 

Passing rates were reviewed from prior 
administrations 
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Policy Review: Level 3 and Level 2 Policy Review: Level 3 and Level 2 


+ Looking at performance on the new Regents + Provide you with small item map that includes 
Exam, we worked backwards and identified: 
o Bookmark pages consistent with policy directive o Range of bookmark locations 
o Narrow range of bookmark locations * Blue for Level 3 

+ When you apply passing rates from older test *-Green fontsvel 2 
to new more rigorous test, the bookmark ° % of students at or above each page 
placements will appear early in the OIB 


o Remember bookmark location is not the same as 
number of points 


These bookmark locations translate to 
reasonable number correct scores 


Policy Review: Level 3 and Level 2 


+ Task Bookmarking Activities 
o What page number that is in line with the Level 4 and Level 5 
policy directive would you recommend be 
used? 
o Please provide your rationale Round 1 
* Knowledge and skills reflected in items 
+ Discussion of PLDs 
+ Expert judgment 
* Impact data 


Start Round 1 Next Steps 


+ Ask yourself the following questions for each * Sign in your OIB when you are done 
threshold 
>» MC Items: Should a student just barely at a threshold be able to 
get this right 2/3 of the time? + Breakfast will be available starting at 7:30 


>» CRlitems: Should a student just barely at a threshold be able to . . 7 
get at least this score point 2/3 of the time? tomorrow morning in the Fitzroy Room 


+ Remember: 
> Threshold Student Descriptions 
> Following order: 
+ Level 3/Level 4 
+ Level 4/Level 5 
> Individual task 


* Meet in this room by 8:30 tomorrow morning 


Day 2 Agenda 


Discuss Round 1 Results 
Break 

Round 2 

Lunch 

Discuss Round 2 Results 
Round 3 

Break 

Discuss Round 3 Results 


Break 


* Meet back here at 10:15 


Lunch 


* Served in Fitzroy 
+ Meet back here at 12:30 


Discuss Round 1 Results 


* Table leader will lead table-level discussions 
for each threshold: 
o Whatis the distribution of bookmark pages? 
o How did you determine your bookmark 
placement? 
+ Use threshold PLD summaries to defend your placement 


Start Round 2 


* Ask yourself: 
» MC Items: Should a just barely Level 4 student be able to get this 
right 2/3 of the time? 
>» CR items: Should a just barely Level 4 student be able to get at 
least this score point 2/3 of the time? 


+ Remember: 
> Threshold Student Descriptions 
» Following order: 
+ Level 3/ Level 4 
+ Level 4/ Level 5 
Individual task 


Discuss Round 2 Results 


* Table leader will lead table-level discussions 
for each threshold: 
o Did the distribution of bookmark pages change? 


o How did you determine your bookmark 
placement? 
+ Use threshold PLD summaries to defend your placement 


* Cross table discussion for each threshold: 
o Are there notable differences across tables 
* Impact data 
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Start Round 3 


* Ask yourself: 
» MC Items: Should a just barely Level 4 student be able to get this 
right 2/3 of the time? 
» CRitems: Should a just barely Level 4 student be able to get at 
least this score point 2/3 of the time? 
+ Remember: 
> Use PLDs 
> Following order: 
+ Level 3/ Level 4 
+ Level 4/Level 5 
> Individual task 
* When complete, sign in your materials and complete 
the evaluation form. 


Thank you! 
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Appendix D: Policy Verification for Level 2 and Level 3 Bookmark 
Placements, Exit Survey and Results, Geometry 


Geometry (Common Core) Level 2/Level 3 and Level 
1/Level 2 Exit Survey Results 


1. | understand the Board of Regents policy directive to place constraints on the overall standard 
setting process, such that the percentage of students who score at Levels 2 and 3 and above on the 
Common Core Regents Exams will remain comparable to those percentages of students who 
scored at a 55 and 65 and above on the current Regents Exams (2005 Standards). 


Percent Selecting Category 


Strongly Moderately Moderate ly Strongly ey g. sD 
Agree Disagree Disagree 
‘ .58 


Strongly Agree = 4, Moderately Agree = 3, Moderately Disagree = 2, Strongly Disagree =1 


2. The impact data (percentages of students at or above the suggested cut scores) presented were 
helpful to me in evaluating the cut scores. 


Percent Selecting Category a 
Strongly Moderately Moderately Strongly 1 
Agree Disagree Disagree AVG: cial 
56.00 20.00 0.00 3.04 .68 


Strongly Agree = 4, Moderately Agree = 3, Moderately Disagree = 2, Strongly Disagree =1 


3. | believe that my Level 2/Level 3 cut score fairly represents the minimal level of achievement for 
students at Level 3, given the policy directive. 


Percent Selecting Category 


Strongly Moderately Moderately Strongly Avg. sD 
Agree Disagree Disagree 
: 57 


Strongly Agree = 4, Moderately Agree = 3, Moderately Disagree = 2, Strongly Disagree =1 
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4. lf you answered Moderately Disagree or Strongly Disagree to Question 3, indicate whether you 


believe the cut score is too high or too low and provide your rationale. 


Percent ae eae Category 
Valid N 


Too | TooHigh § TooLow | Too Low 


2 | 5000 50.00 


Rationale: 


No response 


5. | believe that my Level 2/Level 1 cut score fairly represents the minimal level of achievement for 
students at Level 2, given the policy directive. 


5 Percent Selecting Category 


Valid N Strongly Moderately Moderately Strongly 
Agree Agree Disagree Disagree 


35.71 50.00 10.71 S57 


Strongly Agree = 4, Moderately Agree = 3, Moderately Disagree = 2, Strongly Disagree =1 


6. If you answered Moderately Disagree or Strongly Disagree to Question 5, indicate whether you 
believe the cut score is too high or too low and provide your rationale. 


Percent Selecting Category 
Valid N Too High Too Low 


100.00 0.00 


Rationale: 
NYS Level 2 for congruence says “Identify the image of a given rigid motion”. 


| believe page numbers 4 and 5 reflected the same level (Level 2) so it was difficult to set question 
4 in Level 1. 


Students without the skills to proceed are entering then failing upper math. | have students in Geo 
who can’t do Alg — but passed w/30 cut score — only knowing 30/86 of Algebra. 


Feel that students could not successful pass item #3 at the Level 1. 
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Appendix E: Standard Setting Meeting Exit Survey and Results, Geometry 


Geometry (Common Core) Exit Survey and Results 


1. Please rate the extent of your agreement with each statement regarding the opening 
session: 


Percent Selecting Category 


Strongly 


Valid | Strongly ee 
Disagree 


Avg.’ D 
N Agree ME - 


Disagree 


The opening session provided a 
clear description of the 28 71.42 28.58 0.00 0.00 3.71 .46 
meeting’s goals. 


The opening session helped me 


28 53.57 42.86 3.57 0.00 3.50 58 
understand my tasks. 


The opening session leaders 


clearly explained the 28 57.14 42.86 0.00 0.00 3.57 50 
procedures. 

The opening session addressed 

many of my questions and 28 42.86 57.14 0.00 0.00 3.43 50 
concerns. 


‘Strongly Agree = 4, Agree = 3, Disagree = 2, Strongly Disagree =1 


2. Please rate the extent of your agreement with each statement regarding the Geometry 
training session: 


Percent Selecting Category 
Valid | Strongly i Strongly 
A D 
N Agree ec te Disagree 


The training session leader 
clearly explained the 
procedures. 


64.29 35.71 0.00 


The training session leader 
clearly explained the materials 
used in the bookmark process. 


71.42 28.58 0.00 


The training addressed many of 
my questions and concerns. 


The training materials were 
effective in preparing for 
subsequent tasks. 


The practice exercises were AA.AA 40.74 14.81 
useful. 


‘strongly Agree = 4, Agree = 3, Disagree = 2, Strongly Disagree =1 


42.86 57.14 0 34350 


53.57 46.43 0.00 


| | | 
ihe ereidine Helpedame 28 57.14 42.86 0.00 0.00 3.57 .50 
understand my tasks. 
00 0.00 3.43.50 
27 
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3. Please rate the extent of your agreement with each statement regarding the 
erformance level descriptions (PLDs): 


Percent Selecting Category 
Valid | Strongly ; Strongly 
A D 
N Agree ais Peele Disagree 


Adequate information was 


provided to panelists regarding 40.74 59.26 0.00 
the PLDs. 


Adequate time was provided 
for panelists to gain 55.55 44.44 0.00 
understanding of the PLDs. 


The PLDs communicate a 
reasonable profile of students’ 44.44 51.85 3.70 
achievement at each level. 


‘Strongly Agree = 4, Agree = 3, Disagree = 2, Strongly Disagree =1 


4. Please rate the extent of your agreement with each statement regarding the threshold 
erformance level descriptions (PLDs): 


Percent Selecting Category 
Valid | Strongly : Strongly 
A D 
N Agree oes DEBI aS Disagree 
Adequate time was provided for 


panelists to articulate the 59.26 40.74 0.00 


threshold PLDs. 


The threshold PLDs 
communicate a reasonable 
profile of students’ achievement 
at each threshold. 
‘strongly Agree = 4, Agree = 3, Disagree = 2, Strongly Disagree =1 


48.15 48.15 3.70 
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5. Please indicate your opinion regarding the usefulness of the following materials used: 
Percent Selecting Category 


Valid Very Useful Somewhat Not 
N Useful Useful Useful 


Performance level 60.71 39.29 0.00 0.00 3.61 
descriptions 


53.57 35.71 10.71 0.00 | 3.43 .69 
75.00 25.00 0.00 0.00 
32.14 46.43 14.29 7.14 | 3.04.88 
28.57. 60.71 10.71 0.00 | 3.18 .61 
42.85 39.29 14.29 3.57 | 3.21.83 


‘Very Useful = 4, Useful = 3, Somewhat Useful = 2, Not Useful =1 


6. Please indicate the extent of your satisfaction with the following roles: 
Percent Selecting Category 


Valid Very 7 Partially Not 
N Satisfied Sotishied Satisfied Satisfied 


er aaa 85.71 14.29 0.00 0.00 


DRC room facilitator 78.57 21.43 0.00 0.00 


specialist 


Other DRC Staff 85.19 14.81 0.00 0.00 


Very Satisfied = 4, Satisfied = 3, Partially Satisfied = 2, Not Satisfied =1 
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7. Please indicate your opinion regarding the amount of time allotted for each activity: 


Percent Selecting Category a 


Valid N | Too Little Time About Right Too Much Time 


Training 3.57 : 28.57 


PLD discussion _ 3.57 25.00 


/Roundiratings | 28 | 10.71 ; 14.29 
10.71 17.86 


‘Too Little Time = 1, About Right = 2, Too Much Time = 3 


8. Please indicate the level of confidence you had in placing the bookmark location for 
each assessment cut score: 


Percent Selecting Category 
Valid Very : Partially Not 
N Confident codon Confident Confident 
tevel's/Level¢ 60.71 35.71 3.57 
cut score 
Levella/Levell® 60.71 35.71 3:57 
cut score 


Very Confident = 4, Confident = 3, Partially Confident = 2, Not Confident =1 


9. Please rate the extent of your agreement with each statement regarding the processes 
and results: 


Percent Selecting Category P| 
Valid Strongly : Strongly 1 
A D Avg. D 
N Agree aS oiler Disagree Ne : 


The processes and methods 


used will produce appropriate 40.00 60.00 0.00 : : .50 
results. 


My Bookmark Placements 25 60.00 40.00 0.00 0.00 3.60 .50 
accurately represent the PLDs. 


‘strongly Agree = 4, Agree = 3, Disagree = 2, Strongly Disagree =1 
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